ab-testing

Sets up an A/B test so its result actually means something — what you are testing, how many visitors it needs, and what counts as a win, all decided before it goes live.

WHEN TO USE

SCOPE

USE FOR

Calculate the sample size and test duration needed before launching a test.
Score and prioritize a backlog of experiment ideas by impact, confidence, and ease.
Decide whether a finished test's result is a real win or an inconclusive readout.
Build a growth experimentation program that turns winning tests into a reusable playbook.
Design variant copy or layout changes that isolate a single variable to test.

NOT FOR

Setting up the tracking and event instrumentation a test needs — that's the analytics skill.
Optimizing a page's conversion rate outside of a formal test — that's the cro skill.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (17)

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

Test Context - What are you trying to improve? What change are you considering?
Current State - Baseline conversion rate? Current traffic volume?
Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis

Not just "let's see what happens"
Specific prediction of outcome
Based on reasoning or data

2. Test One Thing

Single variable per test
Otherwise you don't know what worked

3. Statistical Rigor

Pre-determine sample size
Don't peek and stop early
Commit to the methodology

4. Measure What Matters

Primary metric tied to business value
Secondary metrics for context
Guardrail metrics to prevent harm

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Test Types

Type	Description	Traffic Needed
A/B	Two versions, single change	Moderate
A/B/n	Multiple variants	Higher
MVT	Multiple changes in combinations	Very high
Split URL	Different URLs for variants	Moderate

Sample Size

Quick Reference

Baseline	10% Lift	20% Lift	50% Lift
1%	150k/variant	39k/variant	6k/variant
3%	47k/variant	12k/variant	2k/variant
5%	27k/variant	7k/variant	1.2k/variant
10%	12k/variant	3k/variant	550/variant

Calculators:

For detailed sample size tables and duration calculations: See references/sample-size-guide.md

Metrics Selection

Primary Metric

Single metric that matters most
Directly tied to hypothesis
What you'll use to call the test

Secondary Metrics

Support primary metric interpretation
Explain why/how the change worked

Guardrail Metrics

Things that shouldn't get worse
Stop test if significantly negative

Example: Pricing Page Test

Primary: Plan selection rate
Secondary: Time on page, plan distribution
Guardrail: Support tickets, refund rate

Designing Variants

What to Vary

Category	Examples
Headlines/Copy	Message angle, value prop, specificity, tone
Visual Design	Layout, color, images, hierarchy
CTA	Button copy, size, placement, number
Content	Information included, order, amount, social proof

Best Practices

Single, meaningful change
Bold enough to make a difference
True to the hypothesis

Traffic Allocation

Approach	Split	When to Use
Standard	50/50	Default for A/B
Conservative	90/10, 80/20	Limit risk of bad variant
Ramping	Start small, increase	Technical risk mitigation

Considerations:

Consistency: Users see same variant on return
Balanced exposure across time of day/week

Implementation

Client-Side

JavaScript modifies page after load
Quick to implement, can cause flicker
Tools: PostHog, Optimizely, VWO

Server-Side

Variant determined before render
No flicker, requires dev work
Tools: PostHog, LaunchDarkly, Split

Running the Test

Pre-Launch Checklist

During the Test

DO:

Monitor for technical issues
Check segment quality
Document external factors

Avoid:

Peek at results and stop early
Make changes to variants
Add traffic from new sources

The Peeking Problem

Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.

Analyzing Results

Statistical Significance

95% confidence = p-value < 0.05
Means <5% chance result is random
Not a guarantee—just a threshold

Analysis Checklist

Reach sample size? If not, result is preliminary
Statistically significant? Check confidence intervals
Effect size meaningful? Compare to MDE, project impact
Secondary metrics consistent? Support the primary?
Guardrail concerns? Anything get worse?
Segment differences? Mobile vs. desktop? New vs. returning?

Interpreting Results

Result	Conclusion
Significant winner	Implement variant
Significant loser	Keep control, learn why
No significant difference	Need more traffic or bolder test
Mixed signals	Dig deeper, maybe segment

Documentation

Document every test with:

Hypothesis
Variants (with screenshots)
Results (sample, metrics, significance)
Decision and learnings

For templates: See references/test-templates.md

Growth Experimentation Program

Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.

The Experiment Loop

1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat

Hypothesis Generation

Feed your experiment backlog from multiple sources:

Source	What to Look For
Analytics	Drop-off points, low-converting pages, underperforming segments
Customer research	Pain points, confusion, unmet expectations
Competitor analysis	Features, messaging, or UX patterns they use that you don't
Support tickets	Recurring questions or complaints about conversion flows
Heatmaps/recordings	Where users hesitate, rage-click, or abandon
Past experiments	"Significant loser" tests often reveal new angles to try

ICE Prioritization

Score each hypothesis 1-10 on three dimensions:

Dimension	Question
Impact	If this works, how much will it move the primary metric?
Confidence	How sure are we this will work? (Based on data, not gut.)
Ease	How fast and cheap can we ship and measure this?

ICE Score = (Impact + Confidence + Ease) / 3

Run highest-scoring experiments first. Re-score monthly as context changes.

Experiment Velocity

Track your experimentation rate as a leading indicator of growth:

Metric	Target
Experiments launched per month	4-8 for most teams
Win rate	20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses)
Average test duration	2-4 weeks
Backlog depth	20+ hypotheses queued
Cumulative lift	Compound gains from all winners

The Experiment Playbook

When a test wins, don't just implement it — document the pattern:

## [Experiment Name]
**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]

Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.

Experiment Cadence

Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.

Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.

Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.

Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?

Common Mistakes

Test Design

Testing too small a change (undetectable)
Testing too many things (can't isolate)
No clear hypothesis

Execution

Stopping early
Changing things mid-test
Not checking implementation

Analysis

Ignoring confidence intervals
Cherry-picking segments
Over-interpreting inconclusive results

Task-Specific Questions

What's your current conversion rate?
How much traffic does this page get?
What change are you considering and why?
What's the smallest improvement worth detecting?
What tools do you have for testing?
Have you tested this area before?

Related Skills

cro: For generating test ideas based on CRO principles
analytics: For setting up test measurement
copywriting: For creating variant copy

This skill ships inside the Marketing Skills for AI Agents plugin.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add coreyhaines31/marketingskills --skill "ab-testing" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

Install the whole plugin

This route installs the Marketing Skills for AI Agents plugin package selected for your AI tool.

Add the marketplace, then install the plugin

Type both lines inside a Claude Code session, not at your shell prompt. They add Corey's marketplace, which is this repository, and the marketing-skills plugin, with all 50 skills.

/plugin marketplace add coreyhaines31/marketingskills

02
```
/plugin install marketing-skills
```

Installation and loading

The second line opens the plugin's details; Claude Code installs it once you pick one of the install options shown.

Good to know

/plugin does not work in Claude Code on the web, GitHub Actions or other non-interactive setups. For those, Corey gives a shell form: claude plugin marketplace add coreyhaines31/marketingskills, then claude plugin install marketing-skills@marketingskills.

Add the marketplace, then pick the plugin in Codex

Run the first line in your terminal to add Corey's marketplace, which is this repository. Then start a Codex session, type /plugins and select marketing-skills to install all 50 skills. To get new releases later, run codex plugin marketplace upgrade.

codex plugin marketplace add coreyhaines31/marketingskills

02
```
/plugins
```

Open plugin page ↗

What's inside

Files included with this skill.

View on GitHub

Same repository

ad-creative
When the user wants to generate, iterate, or scale ad creative — headlines, descriptions, primary text, or full ad variations — for any paid advertising platform. Also use when the user mentions 'ad copy variations,' 'ad creative,' 'generate headlines,' 'RSA headlines,' 'bulk ad copy,' 'ad iterations,' 'creative testing,' 'write me some ads,' 'Facebook ad copy,' 'Google ad headlines,' 'LinkedIn ad text,' 'static ads,' 'ad templates,' 'iMessage ad,' 'chat reveal ad,' 'ChatGPT ad,' 'Apple Notes ad,' 'AirDrop ad,' 'creative strategy,' 'creative roadmap,' 'creative retro,' 'hook writing,' 'creative review page,' 'present ad creative for approval,' 'motion video ad,' 'faceless video ad,' 'UGC ad,' 'greenscreen ad,' 'TikTok/Reels ad format,' 'which ad format to make,' 'Meta ad format tier list,' or 'creative format taxonomy.' Use this whenever someone needs to produce ad copy at scale or iterate on existing ads. For campaign strategy and targeting, see ads. For landing page copy, see copywriting.Read full description
ads
When the user wants help with paid advertising campaigns on Google Ads, Meta (Facebook/Instagram), LinkedIn, Twitter/X, or other ad platforms. Also use when the user mentions 'PPC,' 'paid media,' 'ROAS,' 'CPA,' 'ad campaign,' 'retargeting,' 'audience targeting,' 'Google Ads,' 'Facebook ads,' 'LinkedIn ads,' 'ad budget,' 'cost per click,' 'ad spend,' 'should I run ads,' 'ABM,' 'account-based marketing,' 'B2B ads,' 'lead quality,' 'negative keywords,' 'Performance Max,' 'thought leader ads,' or 'when should I kill an ad.' Use this for campaign strategy, audience targeting, bidding, and optimization. For bulk ad creative generation and iteration, see ad-creative. For landing page optimization, see cro.Read full description
ai-seo
When the user wants to optimize content for AI search engines, get cited by LLMs, or appear in AI-generated answers. Also use when the user mentions 'AI SEO,' 'AEO,' 'GEO,' 'LLMO,' 'answer engine optimization,' 'generative engine optimization,' 'LLM optimization,' 'AI Overviews,' 'optimize for ChatGPT,' 'optimize for Perplexity,' 'AI citations,' 'AI visibility,' 'zero-click search,' 'how do I show up in AI answers,' 'LLM mentions,' 'optimize for Claude/Gemini,' 'llms.txt,' 'llms-full.txt,' 'OKF,' 'Open Knowledge Format,' 'knowledge bundle,' 'agent-readable site,' 'agent readiness,' 'is my site agent-ready,' 'WebMCP,' 'do listicles still work for AI,' 'ChatGPT stopped citing comparison pages,' or 'AI citation format shift.' Use this whenever someone wants their content to be cited or surfaced by AI assistants and AI search engines. For traditional technical and on-page SEO audits, see seo-audit. For structured data implementation, see schema.Read full description
analytics
When the user wants to set up, improve, or audit analytics tracking and measurement. Also use when the user mentions "set up tracking," "GA4," "Google Analytics," "conversion tracking," "event tracking," "UTM parameters," "tag manager," "GTM," "analytics implementation," "tracking plan," "how do I measure this," "track conversions," "Mixpanel," "Segment," "are my events firing," or "analytics isn't working." Use this whenever someone asks how to know if something is working or wants to measure marketing results. For choosing attribution models, comparing multi-touch/MMM/incrementality, or reconciling conflicting numbers across tools, see attribution. For A/B test measurement, see ab-testing.Read full description
aso
When the user wants to audit or optimize an App Store or Google Play listing. Also use when the user mentions 'ASO audit,' 'app store optimization,' 'optimize my app listing,' 'improve app visibility,' 'app store ranking,' 'audit my listing,' 'why aren't people downloading my app,' 'improve my app conversion,' 'keyword optimization for app,' or 'compare my app to competitors.' Use when the user shares an App Store or Google Play URL and wants to improve it.Read full description
attribution
When the user wants to figure out which marketing actually drives conversions and revenue, choose or interpret an attribution model, or reconcile conflicting numbers across tools. Also use when the user mentions "attribution," "attribution model," "first-touch vs last-touch," "multi-touch," "which channel drives revenue," "what's my real CAC," "my dashboards disagree," "Google/Meta says X but GA says Y," "media mix model," "MMM," "incrementality," "geo lift," "holdout test," "how did you hear about us," "self-reported attribution," "dark social," or wants to instrument attribution themselves — "stitch my bookings to their source," "SavvyCal/Calendly attribution," "close the identify gap," "track conversions on a third-party domain," "first-party / self-hosted attribution." For event tracking setup and UTMs, see analytics. For ad-platform pixels/CAPI, see ads. For pipeline and CRM revenue reporting, see revops. For the AI-search attribution blind spot, see ai-seo.Read full description