Labsco
coreyhaines31 logo

ab-testing

★ 36,000

by coreyhaines31 · part of coreyhaines31/marketingskills

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation...

🔥🔥🔥🔥✓ VerifiedFreeQuick setup
#analytics#git#marketing
🧩 One of 7 skills in the coreyhaines31/marketingskills package — works on its own, and pairs well with its siblings.

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation...

Inspect the full instructions your agent will receiveExpand

This is the exact playbook injected into your agent when the skill activates — shown here so you can audit it before installing. You don't need to read it to use the skill.

Document every test with:

  • Hypothesis

  • Variants (with screenshots)

  • Results (sample, metrics, significance)

  • Decision and learnings

For templates: See references/test-templates.md

Growth Experimentation Program

Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.

The Experiment Loop

Copy & paste — that's it
1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat

Hypothesis Generation

Feed your experiment backlog from multiple sources:

Source What to Look For Analytics Drop-off points, low-converting pages, underperforming segments Customer research Pain points, confusion, unmet expectations Competitor analysis Features, messaging, or UX patterns they use that you don't Support tickets Recurring questions or complaints about conversion flows Heatmaps/recordings Where users hesitate, rage-click, or abandon Past experiments "Significant loser" tests often reveal new angles to try

ICE Prioritization

Score each hypothesis 1-10 on three dimensions:

Dimension Question Impact If this works, how much will it move the primary metric? Confidence How sure are we this will work? (Based on data, not gut.) Ease How fast and cheap can we ship and measure this?

ICE Score = (Impact + Confidence + Ease) / 3

Run highest-scoring experiments first. Re-score monthly as context changes.

Experiment Velocity

Track your experimentation rate as a leading indicator of growth:

Metric Target Experiments launched per month 4-8 for most teams Win rate 20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses) Average test duration 2-4 weeks Backlog depth 20+ hypotheses queued Cumulative lift Compound gains from all winners

The Experiment Playbook

When a test wins, don't just implement it — document the pattern:

Copy & paste — that's it

## [Experiment Name]

**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]

Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.

Experiment Cadence

Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.

Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.

Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.

Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?

Task-Specific Questions

  • What's your current conversion rate?

  • How much traffic does this page get?

  • What change are you considering and why?

  • What's the smallest improvement worth detecting?

  • What tools do you have for testing?

  • Have you tested this area before?

Related Skills

  • cro: For generating test ideas based on CRO principles

  • analytics: For setting up test measurement

  • copywriting: For creating variant copy