diagnosing-experiment-results

The experiment's numbers look wrong and you cannot tell whether to believe them. This walks the known ways a result goes bad and names the one that is actually happening here.

WHEN TO USE

SCOPE

USE FOR

Find out why an experiment shows a warning about bias or an uneven variant split.
Understand why an experiment has 0 exposures despite the feature flag being active.
Make sense of significance that keeps flipping or an A/A test showing a false win.
Reconcile PostHog's experiment numbers with a different count from raw SQL.
Work out what a mid-run change to rollout or metrics did to the results.
Set up a short survey to learn how users felt about the variant they saw.

NOT FOR

Create a brand-new experiment from scratch (use creating-experiments instead).
Change the rollout percentage or variant split rather than diagnose a result (use configuring-experiment-rollout instead).
Configure exposure criteria or metrics rather than diagnose them (use configuring-experiment-analytics instead).
Reset, end, or restart an experiment rather than diagnose its results (use managing-experiment-lifecycle instead).

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (8)

Diagnosing experiment results

This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on?

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged [HIGH], [MEDIUM], or [LOW] based on how strongly it's verified — [HIGH] is verified directly in PostHog code, [MEDIUM] is partially or team-source verified, [LOW] describes SDK/external behavior that wasn't verified here. Treat [LOW] items as hypotheses to test, not facts to assert.

Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the finding-experiments skill first to resolve it to a concrete ID.

Call experiment-get and pull these fields. They are inputs for almost every diagnostic:

parameters.feature_flag_variants[].rollout_percentage — the variant split
parameters.rollout_percentage — the overall rollout (% of users entering the experiment)
exposure_criteria.multiple_variant_handling — defaults to "exclude" if absent
exposure_criteria.exposure_config.event — unset means the default exposure event; read which one from resolved_exposure_event ($feature_flag_called or $experiment_exposure — resolved server-side, same properties either way)
exposure_criteria.filterTestAccounts — defaults to true
feature_flag.active, status (draft / running / paused / exposure_frozen / stopped), start_date, end_date
feature_flag.filters.groups[] — for each group read variant, properties, and rollout_percentage. Any non-null variant is a forced-variant override on the matched cohort (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a variant-pinned group with broad/empty properties at high rollout, or no group left randomized (variant: null) / no release path to one arm — that starves the other variant (one arm gets ~0 analyzable exposures). See references/bias-and-skew.md.
stats_config — Bayesian (default) or Frequentist

Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in references/diagnostic-snapshot.md. Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.

Step 2 — Match symptom to diagnostic

User says...	Diagnostic group
"Smaller variant looks biased" / banner says bias	A — bias & skew
"Variant ratio doesn't match my split" / SRM warning	A — bias & skew
"Why isn't it 50/50?" / "users in both groups"	A — bias & skew
"Users in both control and test" / high `$multiple` %	A — bias & skew
Multi-variant exposure on a server-rendered app	A — bias & skew
Banner about feature-flag/experiment state mismatch	A — bias & skew
"Migrating distinct_id" / "switching from anonymous to user_id" mid-run	A — bias & skew
Metric count is much smaller than exposures (e.g. 10× or 100× gap)	A — bias & skew (route here before D)
"Experiment shows 0 / not enough data" / empty	B — empty experiment
"Variant always undefined / false"	B — empty experiment
"$feature_flag_called fires but no exposures show up"	B — empty experiment
"Experiment says running but exposures haven't moved in weeks/months"	B — empty experiment
"Significance keeps flipping as we run longer"	C — interpretation traps
"Significance was declared, then it wasn't significant anymore"	C — interpretation traps
"30/16 split at 46 exposures, is this broken?"	C — interpretation traps
"A/A test is showing significant results"	C — interpretation traps
"Many metrics — some significant, some not"	C — interpretation traps
"Bayesian says 96% chance to win — should we ship?"	C — interpretation traps
"Confidence intervals overlap — does that mean not significant?"	C — interpretation traps
"An external tool (significance calculator or AI agent) disagrees with PostHog"	C — interpretation traps
"Should I ship? Primary is up but a secondary is down"	C — interpretation traps
"PostHog numbers ≠ my SQL count"	D — numbers vs SQL
"Funnel says X% but my raw event count says Y"	D — numbers vs SQL
"Sum of revenue looks wrong" / "breakdown shows 'none'"	D — numbers vs SQL
"Recordings panel doesn't match the stats"	D — numbers vs SQL
"I applied a filter but the user count didn't change"	D — numbers vs SQL
"I want to slice results by current person properties (as of now, not as of exposure)"	D — numbers vs SQL
"Changed split / rollout / metric / criteria mid-run, now odd"	E — mid-run changes
"Ended/shipped — flag now flipped to 0/100 unexpectedly"	E — mid-run changes
"Long-term metric moves opposite from primary"	E — mid-run changes
"Retention metric counts users I didn't expect"	E — mid-run changes
"Can't convert the feature flag back to a simple (boolean) flag after the experiment ends"	E — mid-run changes
"How do I restart an experiment with new variants?"	E — mid-run changes
Metric line is rendered but the result block is empty / no chance-to-win or significance	E — mid-run changes (E13 legacy methodology)
"Results won't load" / many metric rows show `data: null` (not a legacy experiment)	Step 1.5 — diagnostic snapshot (null rows)
"What do users think of the new flow?" / wants qualitative feedback on an experiment	F — qualitative feedback
"Why did users prefer control?" / "what did they dislike about the test variant?"	F — qualitative feedback

If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes — do not guess.

Step 3 — Surface every diagnostic the evidence supports

After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic that applies before recommending an action.

Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a systematic bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a statistical pattern (e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the bundled advice.

Only list mechanisms that have a path to verification in the project state — config (from experiment-get), snapshot data, activity log, or repo source. Config-derived mechanisms count: an 80/20 split with default multiple_variant_handling="exclude" is visible in experiment-get and is therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean variant ratio) is not.

Diagnostic groups

A — Bias & skew

Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants. Covers the uneven-split + Exclude interaction, SRM, identity fragmentation, bootstrap × /decide mismatch, and flag/experiment state inconsistency.

→ See references/bias-and-skew.md

B — Empty experiment / 0 exposures / "not enough data"

A frequent pain point. Covers SDK call (wrong evaluation method, identify() timing, dedup), exposure capture (custom event missing variant property, required properties, ad-blockers), and exposure-criteria match (test-account filter, eligibility ordering, events firing before exposure).

→ See references/empty-experiment.md

C — Significance / interpretation traps

Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple comparisons, low-volume variance, peeking / early stopping. Includes the legacy stats issue (A/A tests historically over-fired before the new Bayesian module) and how the win-probability methodology changed in Jan 2025 (single test vs control, not control vs all variants).

→ See references/interpretation.md

D — Numbers don't match (PostHog vs the user's SQL / raw count)

The experiment page applies an exposure scope, $multiple exclusion, test-account filter, and date range that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats), breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user confusion, and the recordings-panel-vs-stats divergence.

→ See references/numbers-vs-sql.md

E — Surprises after mid-run changes (incl. lifecycle and retention quirks)

Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding metrics mid-run is p-hacking; ship-variant can rewrite the flag in surprising ways; reset clears results not the flag. Also covers retention-metric quirks (first-event-must-be-after-exposure design), "matured users" filtering, and long-term vs short-term metric divergence.

→ See references/mid-run-changes.md

F — Qualitative feedback: how the change landed, not how far the number moved

Groups A–E find mechanisms. F is for the question they can't reach: what the people in the experiment made of the change. A short survey, shown when users finish the experimented flow, adds that qualitative half — a rating and an optional comment, readable per variant — and works over MCP today.

It suits some experiments and not others. The gate is whether a user could describe the change without being shown both versions: a reworked flow, layout, or process work; a threshold or ranking tweak don't, however large its measured effect. Don't offer it while a mechanical diagnostic is still open — a survey on top of a broken flag gate collects opinions about a feature half the audience never received — and check whether an existing or recent survey already covers the window before proposing a new one.

Two things to get right before creating one: a survey shown to a single variant is itself a difference between the variants, and a running survey linked to the flag can generate exposure events.

→ See references/qualitative-feedback.md

Step 4 — Calibrate recommendations to experiment state

Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the experiment's current state permits.

Draft — config changes are free; recommend and apply.
Running — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe? user-visible?) before recommending. See configuring-experiment-rollout and its reference file references/changing-distribution-after-launch.md for the mid-run rules.
Stopped / archived — the experiment AND its feature flag represent the documented outcome of the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for the next experiment, or (c) explaining what happened.

On a stopped or archived experiment, don't preemptively offer reversal of a state mutation (ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?", explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't signal. Conditional offers like "if this wasn't intended, you could…" or "want me to revert it?" count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?", "can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface reversal mechanics.

Use consistent terminology: variant split (between variants) is distinct from rollout (overall % entering); the default exposure event (resolved_exposure_event) is distinct from a custom exposure event; the Exclude / First seen options control multivariate handling, not exposure.

Related skills

configuring-experiment-analytics — fix the exposure or metric configuration the diagnosis points at
configuring-experiment-rollout — split-change anti-patterns and safe rollout adjustments
managing-experiment-lifecycle — reset, end, or restart when the experiment can't be salvaged in place
analyzing-experiment-session-replays — when the numbers are fine but you need to see the behavior behind them

This skill ships inside the PostHog AI Plugin. Listed package contents include 3 slash commands, 1 agent, hooks and an MCP server. Installed components and skill versions may vary by AI tool.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add posthog/ai-plugin --skill "diagnosing-experiment-results" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

MCP server required

The skill's steps call for tools from the PostHog MCP server. Installing just this skill doesn't add that connection; set it up in your AI tool if it isn't there yet: PostHog MCP setup. Installing the whole plugin instead includes the connection settings.

Requirements for the PostHog MCP server

Connecting is free; sign in with your PostHog account.
Some tools use AI internally and may incur PostHog AI charges. Those tools require AI data processing to be enabled in your organization settings.

Install the whole plugin

This route installs the posthog plugin package selected for your AI tool.

Claude Code

Add PostHog's marketplace, then install the plugin

Run both lines in a terminal. They install PostHog's posthog plugin: this skill, PostHog's other skills, and a connection to PostHog's hosted MCP server.

claude plugin marketplace add PostHog/ai-plugin

02
```
claude plugin install posthog@posthog
```

Installation and loading

The plugin installs for you in every project and loads the next time you start Claude Code, or after /reload-plugins in an open session.
To connect your PostHog account, start claude, type /mcp, select plugin:posthog:posthog and follow the browser prompts.

Good to know

The plugin includes PostHog's session-capture hook; PostHog's README says it sends sessions to LLM Analytics only when POSTHOG_LLMA_CC_ENABLED=true and POSTHOG_API_KEY are both set.
If you use the separate posthog-telemetry plugin, PostHog says to keep only one of the two, because both register the same hook.
On a self-hosted PostHog, this route's MCP connection won't reach your instance: the plugin's config is fixed to https://mcp.posthog.com/mcp, which routes to PostHog's US or EU region. POSTHOG_MCP_URL works only for the Gemini CLI extension.

View on GitHub

Same repository

adding-warehouse-person-properties
Sync columns from a synced data warehouse table onto PostHog person or group properties, so warehouse data becomes usable anywhere person and group properties already work: feature flag targeting, cohorts, insight filters and breakdowns, surveys, session replay filters, workflows, and the person profile. Use when the user wants to "add a person property from my warehouse", "enrich people with Stripe/Postgres/Salesforce data", "put ARR or plan tier on my persons", "target a feature flag by a warehouse column", "sync warehouse columns onto groups or organizations", or wants to inspect, backfill, disable, or debug an existing warehouse-backed person or group property.Read full description
analyzing-expensive-users
Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.Read full description
analyzing-experiment-session-replays
Analyze session replay patterns across experiment variants to understand user behavior differences. Use when the user wants to see how users interact with different experiment variants, identify usability issues, compare behavior patterns between control and test groups, or get qualitative insights to complement quantitative experiment results. Also covers pairing the observed behavior with a linked survey when the user wants qualitative feedback beyond what recordings show.Read full description
assessing-heatmaps
Assesses what a page's heatmap is telling you and recommends concrete changes. Pulls click / rageclick / scroll-depth data for a URL, names the hot elements by cross-referencing autocapture events on the same page, and can create a saved heatmap the user opens in PostHog, then summarizes the behavior and proposes improvements. TRIGGER when: user asks what a heatmap shows, why people aren't clicking something, where users rage-click, how far they scroll, what to change on a page based on heatmap/click data, or to 'analyze/assess/review the heatmap' for a URL. DO NOT TRIGGER when: the user only wants to create a saved heatmap screenshot with no analysis (use heatmaps-saved-create directly), or is asking about session replay in general (use investigating-replay).Read full description
auditing-endpoints
Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks "what endpoints can I clean up?", "are any of my endpoints broken?", "which materialised versions are still being called?", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.Read full description
auditing-experiments-flags
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.Read full description

Diagnosing experiment results

Step 1 — Resolve the experiment

Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Step 2 — Match symptom to diagnostic

Step 3 — Surface every diagnostic the evidence supports

Diagnostic groups

A — Bias & skew

B — Empty experiment / 0 exposures / "not enough data"

C — Significance / interpretation traps

D — Numbers don't match (PostHog vs the user's SQL / raw count)

E — Surprises after mid-run changes (incl. lifecycle and retention quirks)

F — Qualitative feedback: how the change landed, not how far the number moved

Step 4 — Calibrate recommendations to experiment state

Related skills

MCP connection required

Install just this skill

Install the whole plugin

Add PostHog's marketplace, then install the plugin

More skills

adding-warehouse-person-properties

analyzing-expensive-users

analyzing-experiment-session-replays

assessing-heatmaps

auditing-endpoints

auditing-experiments-flags