configuring-experiment-rollout

This skill answers: Who sees what variant?

In most cases, experiments work best with an equal split. If you want to limit exposure to the test variant, adjust the rollout percentage instead.

Why equal splits are better:

• Equal splits maximize statistical power — each variant has the same sample size
• Equal splits balance traffic and thus reach significance faster
• Increasing user exposure throughout the experiment through increasing rollout is clean (changing split mid-experiment can cause users to switch variants, which is bad for user experience and data quality)

Always default to an equal split unless the user explicitly requests otherwise.

Uneven splits combined with the default "Exclude multivariate users" handling can introduce bias. If the experiment observes multi-variant users (users exposed to more than one variant) then those are dropped asymmetrically — the smaller variant loses a larger fraction of its assignments. If those users behave differently from the rest, the smaller variant's metrics will be skewed.

The right mitigation depends on experiment state:

1. Pre-launch, or live but with few exposures so far — use an equal split and reduce the overall rollout. Achieves the same test-variant exposure without the bias and preserves statistical power. See the disambiguation question below.
2. Live experiment with significant exposures — switch multivariate handling to "First seen variant". Changing the split mid-run reassigns users across variants (anti-pattern; see "Changing rollout on a running experiment" below). Switching handling instead keeps everyone in their original variant and avoids the asymmetric exclusion. See configuring-experiment-analytics for how to set this. Note that "first seen" handling can introduce other biases, but it's preferable to mid-run reassignment.

There are two separate controls that determine who sees what. Both live on the linked feature flag, sent through the feature_flag object in the flag's own shape (not the deprecated parameters keys).

CRITICAL: If the user requests an uneven variant split (e.g. "60/40", "70/20/10") or mentions a specific percentage that could refer to either the split or the rollout (e.g. "roll out to 25%"), you MUST clarify before proceeding. This covers two cases:

This option (ensure_experience_continuity on the feature flag) is only relevant when:

• The feature flag is shown to both logged-out AND logged-in users
• You need the same variant assignment before and after login

This is not compatible with all setups. Learn more: https://posthog.com/docs/feature-flags/creating-feature-flags#persisting-feature-flags-across-authentication-steps

Only mention this to the user if their use case involves pre/post-authentication experiences.

Rollout changes require an experiment ID. If the user refers to an experiment by name or description (e.g. "change rollout on my signup test"), load the finding-experiments skill to resolve it to a concrete ID before proceeding.

Any change to rollout or variant split on a running experiment affects both user experience and statistical validity. You MUST warn the user and get explicit confirmation before making the change.

Do NOT silently apply the change — even if the user asked for it directly. Present the warning covering both perspectives:

1. Who sees what variant? — will users switch variants or lose a feature?
2. Who is in my analysis? — how does this affect data quality?

Exception: Increasing rollout (without changing the split) is generally safe — no users switch variants, more users are added cleanly.

If the goal is "stop new users from entering" rather than a percentage change: reducing the rollout is the wrong tool — it drops already-enrolled users out of the experiment too. Freezing exposure (experiment-freeze-exposure) closes enrollment while enrolled users keep their variant and metrics keep flowing; see managing-experiment-lifecycle for its preconditions and limitations.

Mid-experiment fix for uneven-split bias: switching multivariate handling from "Exclude" to "First seen variant" is the recommended mitigation for already-launched experiments — no users switch variants and all collected data stays in the analysis. Changing the split to be even is an anti-pattern mid-run (typically requires resetting or ending the experiment) and is only preferred if the experiment hasn't been exposed to many users yet. See configuring-experiment-analytics for how to change the handling.

See references/changing-distribution-after-launch.md for detailed warnings, what to tell the user, and when to recommend alternatives.

• configuring-experiment-analytics — the analysis side: exposure criteria, metrics, and multivariate handling
• diagnosing-experiment-results — when a mid-run split change has already skewed the results
• managing-experiment-lifecycle — reset or end-and-restart mechanics when a split change requires them